Develop custom robots.txt Parser and Matcher